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Abstract 

Design  of  detectors  for  known  signals  in  non- 
Gausslan  ♦-mixing  noise  is  considered.  Applying 
the  criterion  of  asymptotic  relative  efficiency, 
the  design  of  the  optimal  memoryless  detector  is 
specified  and  is  seen  to  depend  only  on  second- 
order  statistical  knowledge  of  the  noise.  It  is 
then  shown  that  in  many  cases  this  design  reduces 
to  approximating  the  noise  process  with  an  in- 
dependent process  and  obtaining  the  optimal  non- 
linearity through  a limiting  process.  In  addi- 
tion, conditions  are  given  for  the  existence  of 
a unique  optimal  nonlinearity. 

I.  Introduction 

The  notion  of  optimality  of  a detector  has 
been  well  formulated  in  terms  of  the  Neyman- 
Pearson  criterion  [1].  There  is  currently 
increasing  interest  in  the  case  where  the  noise 
is  no-.  -Gaussian  and  dependent;  in  such  a situa- 
tion Neyman-Pearson  techniques  are  frequently 
intractable.  Because  of  this,  it  has  been  found 
necessary  to  adopt  an  alternative  fidelity  cri- 
terion, commonly  that  of  asymptotic  relative 
efficiency  (ARE).  We  consider  memoryless  dis- 
crete time  detection  of  signals  in  corrupting 
noise.  Results  have  been  recently  obtained  [2] 
for  the  case  where  the  noise  is  m-dependent;  the 
resultant  detectors  guarantee  performance  at 
least  as  good  as  that  under  a "white  noise" 
assumption.  The  assumption  of  an  m-dependent 
noise  model  is,  however,  in  several  ways  overly 
restrictive,  and  we  have  therefore  chosen  to 
extend  the  results  of  [2]  to  the  much  broader 
class  of  symmetrically  ♦-mixing  noises.  This 
class  consists  of  processes  which  exhibit  a 
characteristic  rolloff  in  dependency  as  samples 
are  more  widely  separated  in  time,  and  includes 
the  m-dependent  processes. 

II.  Preliminaries 

Let  (N,;  1-1,2,...}  be  a strictly  stationary 
sequence  of  random  variables.  For  a< b,  define 
MCa.bl-olNa.Ng+i Nj,},  the  d-algebra  gene- 

rated by  the  indicated  random  variables,  where 
a and  b may  take  on  extended  real  values.  Then 
{N^;  1-1,2,...}  is  symnetrlcallv  ♦-mixing  if 
there  exists  a nonnegative  sequence  (♦*;  1-1,2, 
...}  with  ♦.-*0  such  that  for  each  k,  l<k<“,  and 
for  each  i>I,  EjeMd.k),  EjCM^+i,-)  together 
imply  ^(E^E^-PCE^PCEz)  |<*t  min{P(E1),P(E2)}. 
We  will  consider  detection  of  a known  constant 
positive  signal  s in  an  additive  symmetrically 
♦-mixing  noise  process,  where  we  observe  real- 
izations {y^;  1-1,..., n}  of  t.ie  process  {Y^; 
i-l,...,n}  and  where  {N2;  1-1,2,...}  is  a sym- 
metrically ♦-mixing  noise  process.  For 


convenience,  we  assume  the  existence  of  Joint 
densities  fj(.,.)  of  Nfc  and  N^+j , with 

* A 11  r-r—r-r- 

K„(x,y)fi  ^ hj(x,y)/ /f(x)f(y)  square  lntegrable 
J"1  A 

for  all  n,  where  hj (x,y)-f j (x,y)+f . (y ,x) , and 
we  will  assume  the  common  univariate  density 
f(-)  is  differentiable  and  strictly  positive  on 
the  real  line  with  finite  Fisher's  information 

number  1(f)  - f(f ' (x)) 2/f (x)  dx  (all  Integrals, 
unless  Indicated  otherwise,  are  taken  over  the 
entire  real  line).  As  in  [2]  we  will  optimize 
over  the  class  of  optimal  memoryless  detectors 
under  a "white  noise"  assumption,  i.e.  where  a 
n 

test  statistic  Tg(y)  - ^g(yj)  is  compared  to 
i-1 

a threshold.  Specifying  the  choice  of  g will 
therefore  be  of  prime  concern. 

We  will  restrict  the  class  © of  nonlineari- 
ties g to  Include  those  measurable  teal-valued 
functions  for  which  we  can  find  sj>0  such  that 
the  random  variable  g(N2+s)  is  second-order  for 
all  se[0,s2],  and  such  that  the  following  mild 
regularity  conditions  hold,  where  sn~K/t/n  for 
some  positive  constant  K and  Es(-)  denotes 
expectation  computed  under  Hi  with  signal 
strength  s (by  proper  choice  of  the  threshold 
T,  we  assume  without  loss  of  generality  that  the 
random  variables  g(Nj)  are  zero  mean) : 


(a)  ^Es{Tg(Y)}ls-0  > 0 for*lln 

(b)  llm^MV^-O^^^O 

n~»  n E0([Tg(Y>r} 

(c)  lim  j g(x)f '(x— sn)dx  - J g(x)f'(x)dx 


(d)  lim 


Eb  {[Tg(Y)-Es  (Tg(Y) } ] } 


n~»  Eo([Tg(Y)]‘) 


(e)  !l/g(x)f(*'s)dx|s-0‘/!j  g(x)f(x-s)  |g_0  d* 


(f)  Oo(g)-E{g(N1)2)  + 2^E(g(N1)g(NJ  + 1))>0. 

J-l 


We  remark  that  the  restrictions  on  the  densi- 
ties fj (•,•).  f(0,  end  the  class  © are  no  more 
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restrictive  than  those  imposed  for  the  m- 
dependent  case  [2].  Properties  (a)-(d)  are 
assumptions  conventionally  Imposed  for  applica- 
tion of  the  Pitman-Noether  theorem,  whereas  (e) 
is  an  exceedingly  mild  restriction.  For  a large 
class  of  noise  processes,  including  all  of  the 
examples  of  [2],  property  (f)  is  automatically 
satisfied  and  may  therefore  be  Ignored. 

From  here  on  we  will  frequently  refer  to 
"Condition  A"  defined  by 

Condition  A:  {N^;  i-1,2,...}  is  symmetrically 
“ , 

♦-mixing  with  < ■>. 

1-1 

2 

The  following  shows  that  oo(g)  is  well-defined 
and  will  allow  us  to  benefit  from  the  asymp- 
totic nature  of  the  ARE  criterion: 

Proposition  1:  Assume  that  Condition  A is 
satisfied  and  that  E{g(Ni)}-0,  E{[g(Nj)]2)  < - 
where  g is  measurable.  Then  (g)  converges 

absolutely,  and  V g(V  ? NCO.q^g)). 

1*1  i/n 

Proof:  The  first  part  of  the  proposition  fol- 
lows as  a direct  corollary  to  Theorem  21.1  of 
[3],  as  does  the  latter  part  if  o^(g)>0.  If 
ofi(g)“0  the  latter  part  follows  as  a consequence 
of  Letmaa  3 of  [3].  QED 


We  are  now  in  position  to  consider  the  pro- 
blem of  determining  necessary  and  sufficient 
conditions  for  the  nonlinearity  g C V to  be 
optimal.  We  first  need  the  following: 

Lemma  1:  Assume  (N^;  1-1,2,...}  is  symmetri- 
cally ♦-mixing  and  suppose  X is  M(l,k)  (M(k+1,«)) 
measurable  and  Y is  respectively  M(k+i,»)  (M(l,k)) 
measurable.  If  h is  a measurable  real-valued 
function  and  Y and  h(X)  second-order,  then 
E { | E { Y | X } — E { Y } | -h(X)  }<2*^(E{Y2))'s(E{h(X)2})1». 

Proof : Consider  the  case  where  X is  M(l,k) 
measurable  and  Y is  M(k+i,«)  measurable.  In  the 

manner  of  [3]  we  may  set  Y - 
Aj  t M(k+i,»).  Then 

|E{Y|X-x}-E{Y)  / fv  I * <y  I x>dy 

j 3 YUj)  *1* 

- / f„(y)dy|. 

j 3 YUj) 

Multiplying  by  h(x)fx(x),  integrating,  and 
applying  the  triangle  and  Schwarz  inequalities 

give  E{|E{Y  | X}  — E { Y } | *h(X)  }<[E{E{Y2  | X}+EI  Y2} ) }*» 

• [ /"^^|P(Aj  | (u»:X(w)-x))-P(Aj)  | •h(x)^fx(x)dx]\ 

J 

whence  comparing  20.27  of  [3]  gives 


^u  I where 

J 3 J 


E{  | E {Y  | X } — E { Y } | *h(Y)  }<2^[E  {Y2  } ]**[£{  (h(X)  )2  } ]**. 

The  other  case  follows  in  a similar  way  using  the 
fact  that  {M^ ; i-1,2,...}  is  symmetrically  $- 
mixing.  QED 

We  can  now  prove  an  important  theorem. 

Theorem  1 : Assume  that  Condition  A is  satisfied 
and  that  g f Then  g is  optimal  (in  the  sense 
of  the  ARE)  if  and  only  if  g is  a solution  (up 
to  a scale  factor)  of 

00 

2 fh,(x.y)  g(y)dy  + g(x)  - f(x)g(x)  (1) 

J-1  3 

where  g(x)  - -f ' (x) . 

Proof : We  consider  the  efficacy  n(g)  - 

es{rCy1)  } | s_0  12/'to(8)  and  its  role  in  the 

ARE  as  given  by  the  Pitman  Noether  theorem  [4]. 
Proceeding  in  the  manner  of  f2]  we  will  obtain 
necessary .and  sufficient  conditions  to  maximize 
H(g)  • -/  g(x) f ' (x)dx  - lo2(g) . Noting  that  we 
are  maximizing  H over  zero-mean  second-order 
g(Nj) , we  may  consider  variations  5 g(N^)  such 
that  (g+e-4g)(Ni)  is  zero-mean  and  second-order, 
hence  5g(Nj)  is  also  zero-mean  and  second-order. 
Defining  Jg(t)  - H(g+t*Sg),  we  then  can  show 

J^(0)  - -J  4g(x)f(*)dx  - 2\J  g(x)6g(x)f(x)dx 

-21  ^ j j [g(y)f  j(x,yHg(y)f  j(y,x)  lSg(x)dydx. 

J-1 

Lemma  1 followed  by  the  Tonelll  and  Fubini 
theorems  implies 


^//[g(y)fig(x)fj(x,y)+g(y)«g(x)fj(y,x)  ] dydx  - 

J-1 


[g(y)f j(*.y)+g(y) fj(y.x) 


6g(x)dx. 


Since  a necessary  condition  for  the  optimality  of 
g is  Jg(0)-0  for  all  admlssable  variations  6g 
and  for  some  1,  we  note  that  varying  1 scales  g, 
and  letting  l-*j,  we  obtain  (1).  In  the  manner 
of  [2],  this  condition  is  seen  also  to  be 
sufficient.  QED 


The  order  of  summation  and  integration  in  (1) 
is  Important  and  is  a noteworthy  example  of  a 
case  when  an  Integral  and  sum  do  not  commute. 
Methods  for  obtaining  the  solution  g are  ad- 
dressed in  the  section  which  follows. 


III.  The  Optimal  Nonlinearity 

As  we  have  remarked,  we  need  consider  solu- 
tions to  (1).  In  the  sequel,  in  view  of  Theorem 
1,  we  will,  up  to  a scale  factor,  identify 
"optimal"  nonllnearitles  with  solutions  g of  (1), 
even  though  one  must  also  check  that  g c V.  To 
approach  the  problem  of  obtaining  g we  need  the 
following: 


- 3 - 


Lemma  2:  Assume  that  Condition  A is  satisfied 
and  ^8n^  is  8 sequence  of  elements  of  ® such 
that  {E{  [gn(Nj^)  ]^}}  is  bounded.  For  n>m  let 

Rmn(x)  “ ^ f (h^ (x.y) /i/f (x) ) gn<y)dy , then 
J“irH 

8up  E{(RmnCN1)/*/f^7)2}  ” 0 . 

n>m 

Proof:  |Rnn(*)/^00|  1^]  |E{gn(Vl)lNrX,l 
j-mfl 


Such  solutions  are  not  in  general  optimal  unless 
the  noise  is  m-dependent  [2],  however  the  g,,, 
will  lead  us  to  the  optimal  nonlinearity  g 
through  convergence  arguments.  Recall  [2]  that 
(5)  possesses  an  associated  family  of  orthonor- 
mal eigenfunctions  { im } ^ and  eigenvalues  { A im } 
We  can  then  show  the  following  second-moment 
property  of  the  {g^: 

Proposition  2:  Assume  that  Condition  A is 
satisfied.  If  -1  is  not  a limit  point  of  the 
double  sequence  then  EKg^Nj))2}  is 

bounded . 


+ £ |E{gn(VlVl*x}l  • <2> 

j-nH-1 

Lemma  1 implies 

E.t|E{gn(N1+l)lNl}|-|E{8n(NJ+l)lNl}l} 

<2^(E{(gn(N1))2})‘1.(E{(E{gn(Nj+1)|N1})2})!5, 

whence  repeated  application  of  Lemma  1 on 
(E{ (E{gn(N^+j) IN^})2})  K-l  times  gives 

K-l  K 

£2”k  ^2~k 

(E{(E{gn(Nj+1)|N1))2}),4<2k’1  ^k’2 

K 

•<E{(gn(N1))2})k_2  •(E{(E{gn(NJ+1)|N1})2})2  . 

Passing  to  the  limit  as  K “ gives 

E{ I E{gn(Ni+l) lNl} l-l E{«n("j+1> 'Nl} I } 


£4^  E{gn(N1)2}. 


Similarly  we  obtain 

E{  | E{gn(N1)  |Ni+1)  | • lEfgnOV  |NJ+1>  | } 

<4^  ^ E(gn(N1)2}. 

In  conjunction  with  the  triangle  inequality, 
(2)-(4)  then  imply 

E{(Rmn(Nl)/'f^V)2^  ^(EfCgnCN!))2))*4. 


We  note  that  {g„(Ni)}  has  bounded  second  moment, 
from  which  the  desired  result  follows.  QED 

Consider  solutions  gg  to 

- ^ Jhj(x,y)gB(y)dy  + g(x)  - f(x)g]>(x).  (5) 

J-l 


Proof : Under  the  hypothesis,  we  may  assume  m 
is  large  enough  so  that  g„,  uniquely  exists  [2], 
Consider  the  solution  g^  to 

- J*Vj(x,y)gB(y)dy  + g(x)  - Af(x)g*(x)  (6) 

j-l 

where  A > 0.  There  exists  arbitrarily  large  A 
such  that  a solution  to  (6)  always  exists 
uniquely.  Multiplying  by  g*(x) , integrating, 
applying  the  Schwarz  inequality  and  Lemma  1 of 
[3]  we  obtain 

4 + [E{(g(N1)/f(N1))2}]*s 

j-l 

• [E{(gV .))2}]'5  > AE  (g^N,))2). 
ml  — m 

X 2 

and  hence  E{(gm(N^))  } is  bounded  over  m for  A 
sufficiently  large  (recall  1(f)  < °°) . The 
solution  of  (6)  satisfies  [5]t 

where  c - /[ g(x)^.  (x)//f  (x)  ]dx  , 


and  therefore 


„ 2 ,2 
c.  A 
im  im 

k 


is  bounded  over  m . 


Defining  S - (i:  |A  | <2A}  and  noting  that 
m lm  — 

all  terms  are  nonnegatlve,  we  see  that 
/ . c.  * and  Z.  c.  are  bounded  over  m. 

us  im  im  usc  im 

m m 

But  noting  that  |l+Ain|  ^ e,  we  see  that 
2 ,2 

V Ci®  in  1 V 2 , 2 

/ ■ K < ^ Zj  C A and 

iTs  (l+A.  )2  “ e2  irS  im  in 

m im  m 


C2  A2  \2 

Elm  im  [ 2X 1 ^ 

c (WJ2  " VXV  S 


cim  ‘ S*nce  121 • 


- 4 - 


*-<x)  - M - 


g(x)  1 "im  'ini  „ , , Consider  solutions  z to 

/f(*>  j.i  lm  -JK  (x,y)zSy)  >/f(y)dy-R  (x)-Xz  (x)  ^(x) . 

I*  j m pni  nm  nm 


obtain  the  desired  result  from  the  triangle 
inequality.  Qp.p 

An  important  relationship  between  the  { gm) 
and  the  optimal  nonlinearity  g may  be  seen  in 
the  following  (where  from  here  on,  unless  stated 
otherwise,  the  ^ are  regarded  as  operating  on 


?ropo8ltlon  3:  Assume  that  Condition  A is 
satisfied.  If  a subsequence  (e  ) of 

“k  k-1 


l«mVl 


satisfies 


g,  then  g is  optimal. 


Proof : We  have  that  g^  satisfies 
0 “ ~fs g(x)f'(x)dx  - 2xjg  (x)Sg(x)f (x)dx 


- 2X  S^y)  hj(x»y){8(x)dxdy  (7) 

J-l  ^ 

where  5g(Nj)  is  a zero  mean,  second-order  varia- 
tion. Application  of  the  Schwarz  inequality 
gives 

lim  f g_  (x)*g(x)dx  - f g(x)5g(x)f (x)dx. 

V-H.  J “k  J 


Thus  z is  a solution  to 
nm 


- ]£/yx*y>I,L(y)dy  _ J2  /Vx’ y>8n<y)«iy 

j“l  JnH-1 

* If  we  multiply  each  side  of  the 

above  by  Zqd(x)  and  Integrate,  Lemma  1 of  [3] 
in  conjunction  with  Proposition  2 imply,  for 
large  fixed  X,  sup  E((zX  (N,))2}  -*■  0. 
nxn  1 


„(*>,  (9) 


But  (8)  implies  “ 

kt-y  X Rmn(x)  1 VdimnAim 

2nm(x)-  ~X X “XTT * 

i-1  im 

where  d,  - f (-R (x) ) (x)dx. 

inn  J inn  m 


Lemma  2 implies  sup  o?  X^  /(X+X  -*■  0 

i-d  imn  im  im 


as  m •>.  We  then  use  the  dominating  techniques 
of  the  proof  of  Proposition  2 to  obtain 


8^EdLnAL/(l+Ai,/"oa8^," 


Application  of  the  triangle  inequality  with 
Lemma  1 of  [3]  shows 


gm^y)  hj (x,y)«g(*)dxdy 


y*  ffe( y)  hj  (x  ,y)  Sg(x)  dxdy . 

3*1 

The  result  follows  now  from  the  proof  of  Theorem 
1 by  taking  limits  of  both  sides  of  (7) . QED 

The  following  provides  conditions  for  the  exis- 
tence of  a unique  optimal  nonlinearity  and 
specifies  how  it  is  related  to  the  {g^} t 

Proposition  4:  Assume  that  Condition  A is 
satisfied.  If  -1  is  not  a limit  point  of 
(Xim}i  m then  there  exists  a unique  (up  to  a 
scale  factor)  optimal  nonlinearity  g,  and 

Proof : Note  that  the  consents  prefacing  the 
proof  of  Proposition  2 apply  here.  For  gn  and 
ia  with  n>m,  setting  *nm-8n"8m  glve8 

-J  K*(x,y)  z^y)  (y)dy-Ran(x)-znn(x)  h (x)  . 


applying  (9)  with  \ - 1 in  conjunction  with 

Lemma  2 gives  sup  E{(z  (N  ))2}  ■*  0.  In  view 
nm  1 

n>m 

of  Proposition  3,  the  Cauchy  criterion  then 
establishes  existence  and  mean-square  conver- 
gence. An  analogous  argument  for  the  equation 

- hj(x,y)h(y)dy  « f (x)h(x)  shows 

h(N^)  » 0 almost  surely,  from  which  uniqueness 
follows . QED 


IV.  Interpretation  of  Results 

Before  condensing  the  preceding  results,  it 
is  beneficial  to  note  the  corresponding  result 
for  the  m-dependent  case.  The  following  theorem 
is  a summary  of  the  relevant  work  of  Poor  and 
Thomas  [2], 

Theorem  2 : Suppose  the  noise  is  m-dependent. 

We  can  make  the  following  statements: 

(a)  If  all  the  eigenvalues  of  {X.  }”  are  not 

in  i-i 

equal  to  -1,  then  a unique  optimal  nonlinearity 
g„  exists  and  is  given  by 

. (*)  - t&ZL  + fin^im  . 

8“()  f<x>  “J  1+Aim  /fW 

where  clB-/[g(x)i)/ia(x)/f(x)  ]dx  and  g(x)— f'(x). 
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(b)  An  optimal  nonlinearity  t exists  If  and 
only  If  for  any  - -1  we  have  c^B  - 0. 


Existence  and  uniqueness  of  the  optimal  non- 
linearity thus  depends  on  the  Information  con- 
tained In  the  eigenvalues  (i  i . As  we  have 
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seen,  the  same  thing  Is  true  In  the  more  gene- 
ral symmetrically  ♦-mixing  case,  and  we  could 
state  a theorem  analogous  to  Theorem  2 for  Che 
♦-mixing  case  by  Imposing  requirements  on  the 


l.m 


to  insure  that  we  would  have  the 


appropriate  statement  corresponding  to  (b). 

The  difficulty  with  this  is  that  such  a condi- 
tion would  necessarily  involve  relative  conver- 
gence of  the  Cjjg  and  1+X^B  to  zero,  and  would 
therefore  be  extremely  difficult  to  check. 

Since  the  Integral  equation  given  by  Theorem  1 
la  In  most  cases  difficult  to  solve.  It  would 
be  most  useful  to  specialize  our  results  to 
the  case  where  the  solution  can  be  found  as  a 
limit  of  functions  which  are  obtained  from 
standard  Hilbert-Schmldt  techniques.  We  will 
say  that  a nonlinearity  g Is  reachable  If  there 
exists  a subsequence  {g  )“  of  (g)»  which 
“k  k-1  m-1 

converges  In  mean-square  to  g.  Because  the  g^ 
satisfy  (5),  the  closed  form  expressions  of 
[2]  are  applicable;  moreover,  a reachable  non- 
linearity g is  approximately  equal  to  a for 
large  k.  "k 


We  then  may  summarize  our  results  for  the 
symmetrically  ♦-mixing  case  as  follows: 
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Theorem  3:  Assume  that  Condition  A Is  satis- 
fied. We  can  make  the  following  statements: 


(c)  If  -1  Is  not  a limit  point  of  {X  ) 

i ,m 

then  there  exists  a unique  optimal  nonlinearity 
g given  by  gB  m-Li*  g (and  thus  g Is  reachable). 


(d)  There  exists  a reachable  optimal  nonlinear- 
ity g If  and  only  If  g Is  the  mean-square  limit 
of  a subsequence  of  (g_)“  . 

^ m-1 


Proof : This  follows  directly  from  Propositions 
3 and  4.  QED 


V.  Conclusion 

We  have  shown  how  the  optimal  detector  may  be 
designed  for  memoryless  detection  of  signals  In 
a large  class  of  additive  ♦-mixing  noise.  We 
have  seen  that  such  a design  effectively  reduces 
to  detector  design  under  an  asymptotic  in- 
dependent assumption,  whereby  standard  Hilbert- 
Schmldt  techniques  may  be  employed.  This 
design  has  the  further  advantage  of  requiring 
only  second-order  statistical  knowledge  of  the 
noise. 
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